Add node pfail and fail count to cluster info metrics #1910
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Harkrishn Patro <[email protected]>
Signed-off-by: Harkrishn Patro <[email protected]>
hpatro
changed the title
zuiderkwast
reviewed
Apr 3, 2025
Contributor
zuiderkwast
left a comment
zuiderkwast
left a comment
There was a problem hiding this comment.
Makes sense.
So the use case is to be able to write tests more reliably, or is there a "real" use case?
Collaborator
Author
I'm trying to observe first time to node failure detection and time to mark it as complete failure. Without this data, it seems difficult to modify the algorithm and observe the change in behavior. |
Contributor
|
Observability of failure detection. It's a great concept! ;) Yeah it can be useful for users too, not only for us. |
Signed-off-by: Harkrishn Patro <[email protected]>
Collaborator
Author
|
The test seems flaky. Looking at it |
Signed-off-by: Harkrishn Patro <[email protected]>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## unstable #1910 +/- ##
============================================
- Coverage 71.03% 70.99% -0.05%
============================================
Files 123 123
Lines 65682 65721 +39
============================================
- Hits 46660 46656 -4
- Misses 19022 19065 +43
🚀 New features to boost your workflow:
|
zuiderkwast
reviewed
Apr 3, 2025
zuiderkwast
approved these changes
Apr 3, 2025
Contributor
zuiderkwast
left a comment
zuiderkwast
left a comment
There was a problem hiding this comment.
LGTM.
@valkey-io/core-team Please ack ( 👍 ) two new fields in CLUSTER INFO.
madolson
reviewed
Apr 8, 2025
Member
madolson
left a comment
madolson
left a comment
There was a problem hiding this comment.
Do we think having this information is better than just asking end users to run cluster nodes/shards and count the number of failed/pfail nodes? I'm a little worried about end users alarming on this metric, even though it includes nodes that aren't part of quorum and aren't serving any traffic.
Collaborator
Author
With large cluster I would prefer not pulling cluster nodes/shards output and compute this. |
Signed-off-by: Harkrishn Patro <[email protected]>
Collaborator
Author
|
I've added voting nodes pfail/fail as well. If we decouple voting nodes from data serving node (primary) within the same architecture in the future, will have to add two additional metric (primary_fail / primary_pfail). @madolson let me know your thoughts. |
Signed-off-by: Harkrishn Patro <[email protected]>
hwware
approved these changes
Apr 8, 2025
madolson
added
needs-doc-pr
madolson
approved these changes
Apr 14, 2025
madolson
removed
the
needs-doc-pr
madolson
added
the
release-notes
madolson
pushed a commit
to valkey-io/valkey-doc
that referenced
this pull request
Apr 15, 2025
Code changes: valkey-io/valkey#1910 Signed-off-by: Harkrishn Patro <[email protected]>
nitaicaro
pushed a commit
to nitaicaro/valkey
that referenced
this pull request
Apr 22, 2025
New fields in CLUSTER INFO: * `cluster_nodes_pfail` * `cluster_nodes_fail` * `cluster_voting_nodes_pfail` * `cluster_voting_nodes_fail` I'm running few tests and trying to capture partially failed and completely failed count. Slot partially failed / completely failed stats exists but is more difficult to assess the node failure count with that. New output: ``` > CLUSTER INFO cluster_state:fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_nodes_pfail:1 cluster_nodes_fail:0 cluster_voting_nodes_pfail:1 cluster_voting_nodes_fail:0 cluster_known_nodes:3 cluster_size:0 cluster_current_epoch:1 cluster_my_epoch:1 cluster_stats_messages_ping_sent:2104 cluster_stats_messages_pong_sent:1906 cluster_stats_messages_meet_sent:1 cluster_stats_messages_sent:4011 cluster_stats_messages_ping_received:1906 cluster_stats_messages_pong_received:1964 cluster_stats_messages_received:3870 total_cluster_links_buffer_limit_exceeded:0 ``` --------- Signed-off-by: Harkrishn Patro <[email protected]> Signed-off-by: Nitai Caro <[email protected]>
nitaicaro
pushed a commit
to nitaicaro/valkey
that referenced
this pull request
Apr 22, 2025
New fields in CLUSTER INFO: * `cluster_nodes_pfail` * `cluster_nodes_fail` * `cluster_voting_nodes_pfail` * `cluster_voting_nodes_fail` I'm running few tests and trying to capture partially failed and completely failed count. Slot partially failed / completely failed stats exists but is more difficult to assess the node failure count with that. New output: ``` > CLUSTER INFO cluster_state:fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_nodes_pfail:1 cluster_nodes_fail:0 cluster_voting_nodes_pfail:1 cluster_voting_nodes_fail:0 cluster_known_nodes:3 cluster_size:0 cluster_current_epoch:1 cluster_my_epoch:1 cluster_stats_messages_ping_sent:2104 cluster_stats_messages_pong_sent:1906 cluster_stats_messages_meet_sent:1 cluster_stats_messages_sent:4011 cluster_stats_messages_ping_received:1906 cluster_stats_messages_pong_received:1964 cluster_stats_messages_received:3870 total_cluster_links_buffer_limit_exceeded:0 ``` --------- Signed-off-by: Harkrishn Patro <[email protected]>
nitaicaro
pushed a commit
to nitaicaro/valkey
that referenced
this pull request
Apr 22, 2025
New fields in CLUSTER INFO: * `cluster_nodes_pfail` * `cluster_nodes_fail` * `cluster_voting_nodes_pfail` * `cluster_voting_nodes_fail` I'm running few tests and trying to capture partially failed and completely failed count. Slot partially failed / completely failed stats exists but is more difficult to assess the node failure count with that. New output: ``` > CLUSTER INFO cluster_state:fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_nodes_pfail:1 cluster_nodes_fail:0 cluster_voting_nodes_pfail:1 cluster_voting_nodes_fail:0 cluster_known_nodes:3 cluster_size:0 cluster_current_epoch:1 cluster_my_epoch:1 cluster_stats_messages_ping_sent:2104 cluster_stats_messages_pong_sent:1906 cluster_stats_messages_meet_sent:1 cluster_stats_messages_sent:4011 cluster_stats_messages_ping_received:1906 cluster_stats_messages_pong_received:1964 cluster_stats_messages_received:3870 total_cluster_links_buffer_limit_exceeded:0 ``` --------- Signed-off-by: Harkrishn Patro <[email protected]>
hwware
pushed a commit
to wuranxx/valkey
that referenced
this pull request
Apr 24, 2025
New fields in CLUSTER INFO: * `cluster_nodes_pfail` * `cluster_nodes_fail` * `cluster_voting_nodes_pfail` * `cluster_voting_nodes_fail` I'm running few tests and trying to capture partially failed and completely failed count. Slot partially failed / completely failed stats exists but is more difficult to assess the node failure count with that. New output: ``` > CLUSTER INFO cluster_state:fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_nodes_pfail:1 cluster_nodes_fail:0 cluster_voting_nodes_pfail:1 cluster_voting_nodes_fail:0 cluster_known_nodes:3 cluster_size:0 cluster_current_epoch:1 cluster_my_epoch:1 cluster_stats_messages_ping_sent:2104 cluster_stats_messages_pong_sent:1906 cluster_stats_messages_meet_sent:1 cluster_stats_messages_sent:4011 cluster_stats_messages_ping_received:1906 cluster_stats_messages_pong_received:1964 cluster_stats_messages_received:3870 total_cluster_links_buffer_limit_exceeded:0 ``` --------- Signed-off-by: Harkrishn Patro <[email protected]> Signed-off-by: hwware <[email protected]>
sarthakaggarwal97
pushed a commit
to sarthakaggarwal97/valkey
that referenced
this pull request
Sep 16, 2025
New fields in CLUSTER INFO: * `cluster_nodes_pfail` * `cluster_nodes_fail` * `cluster_voting_nodes_pfail` * `cluster_voting_nodes_fail` I'm running few tests and trying to capture partially failed and completely failed count. Slot partially failed / completely failed stats exists but is more difficult to assess the node failure count with that. New output: ``` > CLUSTER INFO cluster_state:fail cluster_slots_assigned:0 cluster_slots_ok:0 cluster_slots_pfail:0 cluster_slots_fail:0 cluster_nodes_pfail:1 cluster_nodes_fail:0 cluster_voting_nodes_pfail:1 cluster_voting_nodes_fail:0 cluster_known_nodes:3 cluster_size:0 cluster_current_epoch:1 cluster_my_epoch:1 cluster_stats_messages_ping_sent:2104 cluster_stats_messages_pong_sent:1906 cluster_stats_messages_meet_sent:1 cluster_stats_messages_sent:4011 cluster_stats_messages_ping_received:1906 cluster_stats_messages_pong_received:1964 cluster_stats_messages_received:3870 total_cluster_links_buffer_limit_exceeded:0 ``` --------- Signed-off-by: Harkrishn Patro <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New fields in CLUSTER INFO:
cluster_nodes_pfailcluster_nodes_failcluster_voting_nodes_pfailcluster_voting_nodes_failI'm running few tests and trying to capture partially failed and completely failed count. Slot partially failed / completely failed stats exists but is more difficult to assess the node failure count with that.
New output: